fix(fuzzer): Reduce invalid or unsupported test cases in window fuzzer #11902
+46
−17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
|
@kagamiori has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
kagamiori
added a commit
to kagamiori/velox
that referenced
this pull request
Dec 17, 2024
facebookincubator#11902) Summary: WindowFuzzer currently generate many invalid or unsupported test cases. When a test case is invalid, the execution throws in Velox and is not verified against the reference DB. Test cases unsupported by the ReferenceQueryRunner are also not verified against the reference DB. As the result, the test coverage becomes limited. This PR fixes WindowFuzzer to reduce the percentage of invalid and unsupported test cases. Specifically, this PR includes the following fixes and adjustments: 1. When generating partition-by and order-by keys, only use scalar types supported by the ReferenceQueryRunner. 2. Update the type of the row_number column to be INTEGER to match the type of the offset columns of K-Rows frames. 3. Avoid generating NULLs in the offset columns of K-Rows frames. 4. Fail the fuzzer test if less than 50% iterations are verified, either against reference DB or through custom verifiers. Differential Revision: D67360981 Pulled By: kagamiori
kagamiori
force-pushed
the
fix_window_fuzzer_pr
branch
from
5a6aebb to
b4c6ed4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67360981 |
Yuhta
approved these changes
Dec 17, 2024
kagamiori
added a commit
to kagamiori/velox
that referenced
this pull request
Dec 18, 2024
facebookincubator#11902) Summary: WindowFuzzer currently generate many invalid or unsupported test cases. When a test case is invalid, the execution throws in Velox and is not verified against the reference DB. Test cases unsupported by the ReferenceQueryRunner are also not verified against the reference DB. As the result, the test coverage becomes limited. This PR fixes WindowFuzzer to reduce the percentage of invalid and unsupported test cases. Specifically, this PR includes the following fixes and adjustments: 1. When generating partition-by and order-by keys, only use scalar types supported by the ReferenceQueryRunner. 2. Update the type of the row_number column to be INTEGER to match the type of the offset columns of K-Rows frames. 3. Avoid generating NULLs in the offset columns of K-Rows frames. 4. Fail the fuzzer test if less than 50% iterations are verified, either against reference DB or through custom verifiers. Before: ``` I20241217 15:10:26.897787 551721 WindowFuzzer.cpp:535] ==============================> Done with iteration 50 I20241217 15:10:26.942860 551721 AggregationFuzzerBase.cpp:615] Total functions tested: 24 I20241217 15:10:26.942891 551721 AggregationFuzzerBase.cpp:616] Total iterations requiring sorted inputs: 39 (76.47%) I20241217 15:10:26.942906 551721 AggregationFuzzerBase.cpp:618] Total iterations verified against reference DB: 1 (1.96%) I20241217 15:10:26.942916 551721 AggregationFuzzerBase.cpp:620] Total functions not verified (verification skipped / not supported by reference DB / reference DB failed): 22 (43.14%) / 15 (29.41%) / 0 (0.00%) I20241217 15:10:26.942926 551721 AggregationFuzzerBase.cpp:625] Total failed functions: 21 (41.18%) I20241217 15:10:26.942934 551721 WindowFuzzer.cpp:785] Total functions verified in reference DB: 1 ``` After: ``` I20241217 11:47:07.706315 227175 WindowFuzzer.cpp:537] ==============================> Done with iteration 20 I20241217 11:47:07.732537 227175 AggregationFuzzerBase.cpp:616] Total functions tested: 14 I20241217 11:47:07.732566 227175 AggregationFuzzerBase.cpp:617] Total iterations requiring sorted inputs: 16 (76.19%) I20241217 11:47:07.732582 227175 AggregationFuzzerBase.cpp:619] Total iterations verified against reference DB: 9 (42.86%) I20241217 11:47:07.732592 227175 AggregationFuzzerBase.cpp:621] Total functions not verified (verification skipped / not supported by reference DB / reference DB failed): 8 (38.10%) / 0 (0.00%) / 2 (9.52%) I20241217 11:47:07.732602 227175 AggregationFuzzerBase.cpp:626] Total failed functions: 3 (14.29%) I20241217 11:47:07.732610 227175 WindowFuzzer.cpp:788] Total functions verified in reference DB: 6 ``` Reviewed By: Yuhta Differential Revision: D67360981 Pulled By: kagamiori
kagamiori
force-pushed
the
fix_window_fuzzer_pr
branch
from
b4c6ed4 to
8f7e6f4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67360981 |
facebookincubator#11902) Summary: WindowFuzzer currently generate many invalid or unsupported test cases. When a test case is invalid, the execution throws in Velox and is not verified against the reference DB. Test cases unsupported by the ReferenceQueryRunner are also not verified against the reference DB. As the result, the test coverage becomes limited. This PR fixes WindowFuzzer to reduce the percentage of invalid and unsupported test cases. Specifically, this PR includes the following fixes and adjustments: 1. When generating partition-by and order-by keys, only use scalar types supported by the ReferenceQueryRunner. 2. Update the type of the row_number column to be INTEGER to match the type of the offset columns of K-Rows frames. 3. Avoid generating NULLs in the offset columns of K-Rows frames. 4. Fail the fuzzer test if less than 50% iterations are verified, either against reference DB or through custom verifiers. Before: ``` I20241217 15:10:26.897787 551721 WindowFuzzer.cpp:535] ==============================> Done with iteration 50 I20241217 15:10:26.942860 551721 AggregationFuzzerBase.cpp:615] Total functions tested: 24 I20241217 15:10:26.942891 551721 AggregationFuzzerBase.cpp:616] Total iterations requiring sorted inputs: 39 (76.47%) I20241217 15:10:26.942906 551721 AggregationFuzzerBase.cpp:618] Total iterations verified against reference DB: 1 (1.96%) I20241217 15:10:26.942916 551721 AggregationFuzzerBase.cpp:620] Total functions not verified (verification skipped / not supported by reference DB / reference DB failed): 22 (43.14%) / 15 (29.41%) / 0 (0.00%) I20241217 15:10:26.942926 551721 AggregationFuzzerBase.cpp:625] Total failed functions: 21 (41.18%) I20241217 15:10:26.942934 551721 WindowFuzzer.cpp:785] Total functions verified in reference DB: 1 ``` After: ``` I20241217 11:47:07.706315 227175 WindowFuzzer.cpp:537] ==============================> Done with iteration 20 I20241217 11:47:07.732537 227175 AggregationFuzzerBase.cpp:616] Total functions tested: 14 I20241217 11:47:07.732566 227175 AggregationFuzzerBase.cpp:617] Total iterations requiring sorted inputs: 16 (76.19%) I20241217 11:47:07.732582 227175 AggregationFuzzerBase.cpp:619] Total iterations verified against reference DB: 9 (42.86%) I20241217 11:47:07.732592 227175 AggregationFuzzerBase.cpp:621] Total functions not verified (verification skipped / not supported by reference DB / reference DB failed): 8 (38.10%) / 0 (0.00%) / 2 (9.52%) I20241217 11:47:07.732602 227175 AggregationFuzzerBase.cpp:626] Total failed functions: 3 (14.29%) I20241217 11:47:07.732610 227175 WindowFuzzer.cpp:788] Total functions verified in reference DB: 6 ``` Reviewed By: Yuhta Differential Revision: D67360981 Pulled By: kagamiori
kagamiori
force-pushed
the
fix_window_fuzzer_pr
branch
from
8f7e6f4 to
204457b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67360981 |
xiaoxmeng
approved these changes
Dec 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WindowFuzzer currently generate many invalid or unsupported test cases. When a test case is invalid, the execution throws in Velox and is not verified against the reference DB. Test cases unsupported by the ReferenceQueryRunner are also not verified against the reference DB. As the result, the test coverage becomes limited.
This PR fixes WindowFuzzer to reduce the percentage of invalid and unsupported test cases. Specifically, this PR includes the following fixes and adjustments:
Before:
After: